For many applications involving the use of a camera, there is a need for a tool to inform the user of the camera's orientation. An important example of a case where such a tool is needed is when a camera is used to create a mosaic of images covering a wide area where the ground coordinates of every point on the mosaic must be acquired quickly and accurately.
Many applications exist for building panoramic images. Some applications work offline, that is, the mosaic is created after all of the images are taken. This type of method is not always applicable, however, as the mosaic frequently must be created on-line from a live video source, and needs to be updated in real time as the video camera moves. In an online application of this type, the video can theoretically continue forever.
Some applications do allow online creation of a panoramic view from multiple individual photographs or video captures. These applications are usually based on image matching algorithms that compute a matching transformation between two images. FIG. 7 illustrates schematically such a procedure, in which matching transformation T1,2 computes the match between images I1 and I2 (FIG. 7a), in particular point P2 of image I2 corresponding to point P1 of image I1 (FIG. 7b). The accuracy of such algorithms is limited, and certain patterns of camera motion can lead to large errors in the image location as matching errors are accumulated. FIG. 2 illustrates such a case. The accumulated matching error between the first image acquired (20) and the final image acquired (29) can be very large as the composite error is due to errors arising from 10 separate image matching operations.
The possibilities of overcoming this problem are limited, as the mosaic created by the image matching procedure preserves neither the original images from the video nor their associated transformations.
There is thus a long-felt need for a tool that has the following characteristics: (1) it is capable of storing all of the original images collected for the creation of the mosaic in a bank that allows an online mosaic display of these images, and which enables updating of the bank of images and the mosaic display as new images are obtained; (2) the original images stored in the bank can be displayed as a mosaic according to any desired zoom; (3) if many images cover the same area, the images with the proper zoom can be selected for inclusion in the mosaic display, thus allowing the best mosaic view according to the selected zoom; (4) it should always be possible to recall from the bank and to view the original images covering any part of the mosaic; and (5) the mosaic produced by the tool should be free of accumulated stitching errors, yet imposing no limitation on the way the camera is scanning (as in FIG. 2).
Furthermore, for maximum utility, any tool with the above characteristics needs to be able to anchor the images in the bank to the ground in order to provide a rapid and simple way to obtain the ground coordinates of any point in the displayed mosaic or in any of the original images from which the mosaic was formed.
Specifically, when the source of images is a video stream, there is a special need to anchor the last video frame, so that the user can know where the camera is pointing at any given moment. The user can then retrieve the ground coordinates of any point of interest in the video frame.
Several systems exist for ground anchoring of aerial photos. The common method is to allow the user to mark manually several pairs of “tie points.” The first point in the pair is a point on the photo; the second point is marked on a map or orthophoto. A global ground to image transformation is computed, and using this transformation, every ground point can be converted to image point and vice versa. Some systems can create the tie-points automatically. One common way to do it is to use an image to ground matching algorithm (IGA), usually based on matching the current photo to an already anchored photo, usually an orthophoto (FIG. 9). However, good performance by IGAs is not easy to achieve. They frequently fail, especially when trying to match images taken from significantly different view angles, and are in general not sufficiently reliable to be able to provide acceptable results for video anchoring.
Many systems for ground anchoring of aerial photos are generally adapted for offline work. After the photos are taken, the anchoring procedure can be performed either manually or automatically. These systems, however, are not built for anchoring a continuously growing bank of images or mosaic. Specifically, they cannot provide a means for automatically anchoring the last video frame.